Statistical Models with Uncertain Error Parameters

In a statistical analysis in Particle Physics, nuisance parameters can be introduced to take into account various types of systematic uncertainties. The best estimate of such a parameter is often modeled as a Gaussian distributed variable with a given standard deviation (the corresponding "systematic error"). Although the assigned systematic errors are usually treated as constants, in general they are themselves uncertain. A type of model is presented where the uncertainty in the assigned systematic errors is taken into account. Estimates of the systematic variances are modeled as gamma distributed random variables. The resulting confidence intervals show interesting and useful properties. For example, when averaging measurements to estimate their mean, the size of the confidence interval increases for decreasing goodness-of-fit, and averages have reduced sensitivity to outliers. The basic properties of the model are presented and several examples relevant for Particle Physics are explored.Comment: 26 pages, 27 figure

Effect of Systematic Uncertainty Estimation on the Muon g−2g-2 Anomaly

The statistical significance that characterizes a discrepancy between a measurement and theoretical prediction is usually calculated assuming that the statistical and systematic uncertainties are known. Many types of systematic uncertainties are, however, estimated on the basis of approximate procedures and thus the values of the assigned errors are themselves uncertain. Here the impact of the uncertainty {\it on the assigned uncertainty} is investigated in the context of the muon $g-2$ anomaly. The significance of the observed discrepancy between the Standard Model prediction of the muon's anomalous magnetic moment and measured values are shown to decrease substantially if the relative uncertainty in the uncertainty assigned to the Standard Model prediction exceeds around 30\%. The reduction in sensitivity increases for higher significance, so that establishing a $5\sigma$ effect will require not only small uncertainties but the uncertainties themselves must be estimated accurately to correspond to one standard deviation.Comment: 6 pages, 2 figure

Asymptotic formulae for likelihood-based tests of new physics

We describe likelihood-based statistical tests for use in high energy physics for the discovery of new phenomena and for construction of confidence intervals on model parameters. We focus on the properties of the test procedures that allow one to account for systematic uncertainties. Explicit formulae for the asymptotic distributions of test statistics are derived using results of Wilks and Wald. We motivate and justify the use of a representative data set, called the "Asimov data set", which provides a simple method to obtain the median experimental sensitivity of a search or measurement as well as fluctuations about this expectation.Comment: fixed typo in equations 75 & 7

Higher-order asymptotic corrections and their application to the Gamma Variance Model

We present improved methods for calculating confidence intervals and $p$-values in a specific class of statistical model that can incorporate uncertainties in parameters that themselves represent uncertainties (informally, ``errors on errors'') called the Gamma Variance Model (GVM). This model contains fixed parameters, generically called $\varepsilon$, that represent the relative uncertainties in estimates of standard deviations of Gaussian distributed measurements. If the $\varepsilon$ parameters are small, one can construct confidence intervals and $p$-values using standard asymptotic methods. This is formally similar to the familiar situation of a large data sample, in which estimators for all adjustable parameters have Gaussian distributions. Here we address the important case where the $\varepsilon$ parameters are not small and as a consequence the asymptotic distributions do not represent a good approximation. We investigate improved test statistics based on the technology of higher-order asymptotics ($p^*$ approximation and Bartlett correction).Comment: 22 pages, 8 figure

Comparison of unfolding methods using RooFitUnfold

In this paper we describe RooFitUnfold, an extension of the RooFit statistical software package to treat unfolding problems, and which includes most of the unfolding methods that commonly used in particle physics. The package provides a common interface to these algorithms as well as common uniform methods to evaluate their performance in terms of bias, variance and coverage. In this paper we exploit this common interface of RooFitUnfold to compare the performance of unfolding with the Richardson-Lucy, Iterative Dynamically Stabilized, Tikhonov, Gaussian Process, Bin-by-bin and inversion methods on several example problems